Papers with dense counterpart
Learning Sparsity for Effective and Efficient Music Performance Question Answering (2025.acl-short)
Copied to clipboard
| Challenge: | Existing Music AVQA methods rely on dense and unoptimized representations, leading to inefficiencies in the isolation of key information, reduction of redundancy, and prioritization of critical samples. |
| Approach: | They propose a sparse learning framework specifically designed for Music AVQA to address these challenges. |
| Outcome: | The proposed framework reduces training time by 28.32% while maintaining accuracy while maintaining state-of-the-art performance on the Music AVQA datasets. |
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval (2021.naacl-main)
Copied to clipboard
| Challenge: | SPARTA is a novel neural retrieval method for open-domain question answering . it learns a sparse representation that can be efficiently implemented as an Inverted Index . |
| Approach: | They propose a method that learns a sparse representation that can be implemented as an Inverted Index. |
| Outcome: | The proposed method achieves state-of-the-art results on 4 open-domain question answering tasks and 11 retrieval question answering (ReQA) tasks. |
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024.acl-long)
Copied to clipboard
Damai Dai, Chengqi Deng, Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, Wenfeng Liang
| Challenge: | Mixture-of-Experts (MoE) architectures face challenges in ensuring expert specialization . despite the promising performance, scaling language models to an extremely large scale is associated with exceedingly high computational costs. |
| Approach: | They propose an architecture that allows for ultimate expert specialization by segmenting experts into mN ones and activating mK from them. |
| Outcome: | The proposed architecture achieves comparable performance with GShard with 2B parameters and computation. |
CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling (2025.findings-emnlp)
Copied to clipboard
Xinze Wang, Chen Chen, Yinfei Yang, Hong-You Chen, Bowen Zhang, Aditya Pal, Xiangxin Zhu, Xianzhi Du
| Challenge: | Mixture-of-Experts (MoE) models are crucial for scaling model capacity while controlling inference costs. |
| Approach: | They propose an alternative training strategy that converts a dense CLIP model into a sparse MoE architecture. |
| Outcome: | The proposed training strategy outperforms dense models on COCO and Flickr30k benchmarks. |
Mixture of Languages: Improved Multilingual Encoders Through Language Grouping (2025.emnlp-main)
Copied to clipboard
João Maria Janeiro, Belen Alastruey, Francisco Massa, Maha Elbayad, Benjamin Piwowarski, Patrick Gallinari, Loic Barrault
| Challenge: | Recent work in this field relies on training transformer encoders on a large amount of multilingual data, with all parameters shared across all languages. |
| Approach: | They propose a mixture of languages strategy to pretrain largely multilingual encoders using masked language modeling. |
| Outcome: | The proposed architecture outperforms a dense counterpart, MoE models and public multilingual encoders on downstream tasks while minimizing interference. |